Conversation
metra
left a comment
metra
left a comment
There was a problem hiding this comment.
Approving with some comments. I don't have much context to review this.
| // via the k8s application, like in cluster mode driver | ||
| childClasspath ++= resolvedMavenCoordinates.split(",") | ||
| } else { | ||
| // In K8s client mode, when in the driver, add resolved jars early as we might need |
| for (jar <- resolvedMavenCoordinates.split(",")) { | ||
| addJarToClasspath(jar, loader) | ||
| } | ||
| } |
There was a problem hiding this comment.
I'm not sure why you added if (isKubernetesClusterModeDriver) to the else branch but I presume it's for a good reason.
There was a problem hiding this comment.
It's because when spark-submit is in the k8s driver - deployMode == client
val isKubernetesCluster = clusterManager == KUBERNETES && deployMode == CLUSTER
val isKubernetesClient = clusterManager == KUBERNETES && deployMode == CLIENT
val isKubernetesClusterModeDriver = isKubernetesClient &&
sparkConf.getBoolean("spark.kubernetes.submitInDriver", false)
python/pyspark3/version.py
Outdated
| # limitations under the License. | ||
|
|
||
| __version__ = "3.1.2+affirm4" | ||
| __version__ = "3.1.2+affirm8" |
| <artifactId>jackson-dataformat-yaml</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> |
| return serviceHost != null && serviceHost.length > 0 | ||
| } | ||
|
|
||
| def getHomeDir(): String = { |
There was a problem hiding this comment.
Not sure what sets it, but it's always present on k8s containers. We have something similar in ml_pipelines
There was a problem hiding this comment.
I meant more are you writing this code from scratch or are you porting it from somewhere?
There was a problem hiding this comment.
Oh it's coming from @daggertheog's diff for the spark 2 k8s auth issue
2 participants
What changes were proposed in this pull request?